Skip to content

feat(experimentation): environment-scoped metrics & experiment results#7674

Merged
Zaimwa9 merged 10 commits into
mainfrom
feat/experiment-metrics
Jun 8, 2026
Merged

feat(experimentation): environment-scoped metrics & experiment results#7674
Zaimwa9 merged 10 commits into
mainfrom
feat/experiment-metrics

Conversation

@gagantrivedi

@gagantrivedi gagantrivedi commented Jun 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Member
  • I have read the Contributing Guide.
  • I have added information to docs/ if required so people know about the feature.
  • I have filled in the "Changes" section below.
  • I have filled in the "How did you test this code" section below.

Changes

Adds an environment-scoped, reusable Metric and the API to attach metrics to experiments. Data-model + API

  • Metric — env-scoped, soft-delete: name, description, aggregation (count/sum/mean/occurrence), JSON definition. Immutable (no update).
  • ExperimentMetric — join with expected_direction; unique per (experiment, metric).
  • API (gated on experimental_flags + env admin):
    • …/environments/{key}/experiment-metrics/ — list/create/retrieve/delete (delete blocked 409 while attached to an active experiment).
    • …/experiments/{id}/metrics/ — attach/list/detach, with same-environment + unique-attach validation.

How did you test this code?

Unit tests under api/tests/unit/experimentation/ (models, library CRUD, attach/detach, gating, validation). Full experimentation suite green (181 passed); mypy/ruff/flagsmith-lint-tests clean.

DATABASE_URL=… uv run --directory api pytest tests/unit/experimentation/

Manual (flag on, as env admin): create a metric → 201; attach to an experiment → 201, re-attach same metric → 400; PATCH a metric → 405; delete while attached → 409, detach then delete → 204.

@gagantrivedi gagantrivedi requested review from a team as code owners June 2, 2026 09:50
@gagantrivedi gagantrivedi requested review from emyller and removed request for a team June 2, 2026 09:50
@vercel

vercel Bot commented Jun 2, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
flagsmith-frontend-preview Ready Ready Preview, Comment Jun 8, 2026 8:11am
flagsmith-frontend-staging Ready Ready Preview, Comment Jun 8, 2026 8:11am
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Ignored Ignored Preview Jun 8, 2026 8:11am

Request Review

@github-actions github-actions Bot added api Issue related to the REST API infrastructure feature New feature or request and removed infrastructure labels Jun 2, 2026
@github-actions

github-actions Bot commented Jun 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Docker builds report

Image Build Status Security report
ghcr.io/flagsmith/flagsmith-api-test:pr-7674 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-e2e:pr-7674 Finished ✅ Skipped
ghcr.io/flagsmith/flagsmith-frontend:pr-7674 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith-api:pr-7674 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith:pr-7674 Finished ✅ Results ✅
ghcr.io/flagsmith/flagsmith-private-cloud:pr-7674 Finished ✅ Results ✅

@gagantrivedi gagantrivedi marked this pull request as draft June 2, 2026 09:53
@gagantrivedi gagantrivedi removed the request for review from emyller June 2, 2026 09:53
@gagantrivedi gagantrivedi assigned emyller and unassigned emyller Jun 2, 2026
@codecov

codecov Bot commented Jun 2, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.54%. Comparing base (d0ac9b5) to head (282c32f).

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #7674      +/-   ##
==========================================
+ Coverage   98.38%   98.54%   +0.16%     
==========================================
  Files        1446     1452       +6     
  Lines       55321    55711     +390     
==========================================
+ Hits        54427    54901     +474     
+ Misses        894      810      -84

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jun 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  33 seconds
commit  86ccf2a
info  🔄 Run: #17168 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  35.1 seconds
commit  86ccf2a
info  🔄 Run: #17168 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  39.8 seconds
commit  86ccf2a
info  🔄 Run: #17168 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  52.5 seconds
commit  86ccf2a
info  🔄 Run: #17168 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  40.7 seconds
commit  1fd7efb
info  🔄 Run: #17290 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  37.5 seconds
commit  1fd7efb
info  🔄 Run: #17290 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  32.9 seconds
commit  1fd7efb
info  🔄 Run: #17290 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  39.7 seconds
commit  1fd7efb
info  🔄 Run: #17290 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  33 seconds
commit  de3e8b9
info  🔄 Run: #17291 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  43 seconds
commit  de3e8b9
info  🔄 Run: #17291 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  34.2 seconds
commit  de3e8b9
info  🔄 Run: #17291 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  39.9 seconds
commit  de3e8b9
info  🔄 Run: #17291 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  32.9 seconds
commit  68e8b97
info  🔄 Run: #17311 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  38.2 seconds
commit  68e8b97
info  🔄 Run: #17311 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  32.4 seconds
commit  68e8b97
info  🔄 Run: #17311 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  4 passed

Details

stats  4 tests across 4 suites
duration  57 seconds
commit  68e8b97
info  🔄 Run: #17311 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  40.6 seconds
commit  282c32f
info  🔄 Run: #17318 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

passed  1 passed

Details

stats  1 test across 1 suite
duration  45.2 seconds
commit  282c32f
info  🔄 Run: #17318 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

passed  3 passed

Details

stats  3 tests across 3 suites
duration  33.8 seconds
commit  282c32f
info  🔄 Run: #17318 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

passed  2 passed

Details

stats  2 tests across 2 suites
duration  43.5 seconds
commit  282c32f
info  🔄 Run: #17318 (attempt 1)

@github-actions

github-actions Bot commented Jun 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Visual Regression

19 screenshots compared. See report for details.
View full report

@gagantrivedi gagantrivedi force-pushed the feat/experiment-metrics branch from 86ccf2a to 435d91f Compare June 2, 2026 10:41
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request labels Jun 2, 2026
@gagantrivedi gagantrivedi force-pushed the feat/experiment-metrics branch from 435d91f to 9568226 Compare June 2, 2026 11:01
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request labels Jun 2, 2026
@gagantrivedi gagantrivedi force-pushed the feat/experiment-metrics branch from 9568226 to 2fea3fa Compare June 3, 2026 07:00
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request labels Jun 3, 2026
@gagantrivedi
feat(experimentation): environment-scoped metrics & experiment-metric…
c961486 
… attachment

Add a reusable, environment-scoped Metric and the ExperimentMetric join that
attaches metrics to experiments.

- Models: Metric (numeric; count/sum/mean/occurrence aggregations + JSON
  definition), ExperimentMetric (expected_direction; one attach per
  experiment+metric); Experiment gains exposure_event ($flag_exposure) and
  control_variant.
- Metric library CRUD under environments/{key}/experiment-metrics/, gated on
  EXPERIMENT_FLAG + environment admin. Metrics are immutable for now (no
  update); deletion blocked while attached to an active experiment.
- Attach/detach metrics under an experiment, with same-environment and
  unique-attach validation.

Results computation (ClickHouse query builder + statistics) is intentionally
kept on a separate branch; this branch is models + API only.
@gagantrivedi gagantrivedi force-pushed the feat/experiment-metrics branch from 2fea3fa to c961486 Compare June 3, 2026 07:21
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request labels Jun 5, 2026
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request labels Jun 5, 2026
@Zaimwa9 Zaimwa9 requested review from a team as code owners June 8, 2026 07:34
@Zaimwa9 Zaimwa9 requested review from talissoncosta and removed request for a team June 8, 2026 07:34
@github-actions github-actions Bot added front-end Issue related to the React Front End Dashboard docs Documentation updates labels Jun 8, 2026
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request docs Documentation updates labels Jun 8, 2026
@Zaimwa9 Zaimwa9 removed the request for review from talissoncosta June 8, 2026 08:10
@github-actions github-actions Bot added the docs Documentation updates label Jun 8, 2026
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request docs Documentation updates labels Jun 8, 2026
matthewelwell
matthewelwell approved these changes Jun 8, 2026
View reviewed changes

@matthewelwell matthewelwell left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving it based on @gagantrivedi 's approval.

@flagsmith-engineering flagsmith-engineering Bot mentioned this pull request Jun 8, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Zaimwa9 Zaimwa9 Zaimwa9 left review comments

Copilot code review Copilot Copilot left review comments

@matthewelwell matthewelwell matthewelwell approved these changes

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

api Issue related to the REST API feature New feature or request front-end Issue related to the React Front End Dashboard

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@gagantrivedi @Zaimwa9 @matthewelwell @emyller @talissoncosta